On dangers of cross-validation in steganalysis

Modern steganalysis is a combination of a feature space design and a supervised binary classification. In this report, we assume that the feature space has been already constructed, i.e., the steganalyst has a set of training features and needs to train a binary classifier. Any machine learning tool can be used for this task and its parameters can be tuned through cross-validation, a standard automated model-selection procedure. However, classification problems arising in steganalysis have a very specific nature – individual training samples naturally form pairs of cover–stego feature vectors with opposite labels lying close to each other in the feature space. It is important to preserve these cover–stego pairs during cross-validation (prevent splitting each pair into different folds) otherwise the obtained error estimates may be misleading and lead to a suboptimal performance of the classifier. In this report, we demonstrate the sketched problem with cross-validation on a specific example of image steganalysis in the JPEG domain. As a classifier, we selected the support vector machine (SVM), a popular choice in steganalysis. In particular, we show that the implicit k-fold cross-validation as implemented in LIBSVM [2], a widely used implementaion of SVM, is not suitable for steganalysis and may result in a suboptimal performance and a striking discrepancy between the predicted and the real testing error. Instead of the implicit k-fold cross-validation, a steganalysis-aware cover-stego pair preserving cross-validation should be used. We stress that this is a steganalysis-specific issue and does not indicate any implementation flaw in LIBSVM. The issue with the standard cross-validation procedure in steganalysis has already been pointed out by Schwamberger and Franz in 2010 [14]. We believe, however, that the message may have been hidden to the reader in other experiments and conclusions presented in [14], as authors studied not only the cross-validation, but also different normalization techniques, and performed numerous experiments using different features and stego-algorithms. This technical report, on the other hand, is devoted solely to the problem of improper cross-validation. We go more in depth, provide explanation, and also study severity w.r.t. payload. Moreover, we point out a few examples of published work with results affected by the improper cross-validation.